Efficient HTTP based I/O on very large datasets for high performance computing with the libdavix library

摘要

Remote data access for data analysis in high performance computing iscommonly done with specialized data access protocols and storage systems. Theseprotocols are highly optimized for high throughput on very large datasets,multi-streams, high availability, low latency and efficient parallel I/O. Thepurpose of this paper is to describe how we have adapted a generic protocol,the Hyper Text Transport Protocol (HTTP) to make it a competitive alternativefor high performance I/O and data analysis applications in a global computinggrid: the Worldwide LHC Computing Grid. In this work, we first analyze thedesign differences between the HTTP protocol and the most common highperformance I/O protocols, pointing out the main performance weaknesses ofHTTP. Then, we describe in detail how we solved these issues. Our solutionshave been implemented in a toolkit called davix, available through severalrecent Linux distributions. Finally, we describe the results of our benchmarkswhere we compare the performance of davix against a HPC specific protocol for adata analysis use case.